Image processing device, imaging device, microscope device, image processing method, and image processing program

ABSTRACT

The image processing device 10 includes a template preparation unit 15 for preparing, from a template included in pixels of M rows and M columns (M is an integer not less than 3) corresponding to a molecular model, a partial template corresponding to a shape for which a shape of the molecular model is divided, an evaluation value calculation unit 17 for evaluating, in the optical image, by use of the partial template, matching between the optical image and the partial template to calculate an evaluation value for every plurality of the attention pixels, and a molecular location identification unit 18 for identifying the molecular location in the optical image based on the evaluation value.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Provisional Application Ser. No.61/670,264 filed on Jul. 11, 2012 and Japanese Patent Application No.2011-288390 filed on Dec. 28, 2011, which is hereby incorporated byreference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing device, imagingdevice, microscope device, image processing method, and image processingprogram for identifying a molecular location from an image picture of asample.

Related Background Art

Conventionally, it has been tried to image light from a samplecontaining fluorescent molecules or the like, and identify the locationof the molecules such as fluorescent molecules by means of image dataaccordingly obtained. For example, the following Non Patent Literature 1discloses a method for preparing a super-resolution image from anoptical image of a sample. In this method, a pixel having the maximumluminance value among the pixels composing an optical image isidentified as a candidate molecule, and after fitting a luminancedistribution of model molecules to the image, and then subtracting afitted model molecule luminance distribution from the image, theabove-described candidate molecule is stored. Then, for the image aftersubtraction, identification of a candidate molecule, fitting,subtraction of a luminance distribution from an image, and storing ofthe candidate molecule is repeated, and a super-resolution image isgenerated based on the stored candidate molecules.

-   Non Patent Literature 1: Seamus J Holden, Stephan Uphoff & Achillefs    N Kapanidis, “DAOSTORM: an algorithm for high density    super-resolution microscopy,” Nature Methods, Vol. 8, No. 4, April    2011

SUMMARY OF THE INVENTION

However, the method described in Non Patent Literature 1 described aboveis applicable when a plurality of molecules are distant from each other,while when a plurality of molecules are at close distance from eachother, an optical image is generated with light from the respectivemolecules overlapped, and thus in the method for identifying a candidatemolecule based on the luminance value, it is difficult to distinguish aplurality of molecules separately. Alternatively, a plurality ofmolecules may be misidentified as a single molecule.

Therefore, the present invention has been made in view of such problems,and it is an object thereof to provide an image processing device,imaging device, microscope device, image processing method, and imageprocessing program capable of identifying a plurality of molecules withhigh accuracy.

In order to solve the above-described problems, one aspect of an imageprocessing device of the present invention is an image processing devicefor identifying a molecular location based on an image picture of asample obtained by an imaging element, and includes a template preparingmeans for preparing, from a template included in pixels of M rows and Mcolumns (M is an integer not less than 3) corresponding to a molecularmodel, a partial template corresponding to a shape for which a shape ofthe molecular model is divided into a predetermined ratio, an evaluationvalue calculating means for evaluating, in the image picture, by use ofthe partial template corresponding to shapes rotated by a predeterminedangle each about a selected attention pixel, matching between the imagepicture and the partial template to calculate an evaluation value forevery plurality of the attention pixels, and a molecular locationidentifying means for identifying the molecular location in the imagepicture based on the evaluation value calculated for a plurality of theattention pixels in the image picture.

Alternatively, one aspect of an imaging device of the present inventionincludes the foregoing image processing device, and an imaging elementfor obtaining the image picture.

Alternatively, one aspect of a microscope device of the presentinvention includes the foregoing image processing device, an imagingelement for obtaining the image picture, and an optical system forgenerating an image of the sample for the imaging element.

Alternatively, one aspect of an image processing method of the presentinvention is an image processing method for identifying a molecularlocation based on an image picture of a sample obtained by an imagingelement, and includes a template preparing step of an image processingdevice preparing, from a template included in pixels of M rows and Mcolumns (M is an integer not less than 3) corresponding to a molecularmodel, a partial template corresponding to a shape for which a shape ofthe molecular model is divided into a predetermined ratio, an evaluationvalue calculating step of the image processing device evaluating, in theimage picture, by use of the partial template corresponding to shapesrotated by a predetermined angle each about a selected attention pixel,matching between the image picture and the partial template to calculatean evaluation value for every plurality of the attention pixels, and amolecular location identifying step of the image processing deviceidentifying the molecular location in the image picture based on theevaluation value calculated for a plurality of the attention pixels inthe image picture.

Alternatively, one aspect of an image processing program of the presentinvention is an image processing program for identifying a molecularlocation based on an image picture of a sample obtained by an imagingelement, and causes a computer to function as a template preparing meansfor preparing, from a template included in pixels of M rows and Mcolumns (M is an integer not less than 3) corresponding to a molecularmodel, a partial template corresponding to a shape for which a shape ofthe molecular model is divided into a predetermined ratio, an evaluationvalue calculating means for evaluating, in the image picture, by use ofthe partial template corresponding to shapes rotated by a predeterminedangle each about a selected attention pixel, matching between the imagepicture and the partial template to calculate an evaluation value forevery plurality of the attention pixels, and a molecular locationidentifying means for identifying the molecular location in the imagepicture based on the evaluation value calculated for a plurality of theattention pixels in the image picture.

According to such an image processing device, imaging device, microscopedevice, image processing method, and image processing program, a partialtemplate divided from a template included in pixels of M rows and Mcolumns is prepared, and by use of a partial template corresponding to ashape rotated about an attention pixel selected from an image picture,matching between the partial template and image picture is evaluated,and a molecular location is identified based on the obtained evaluationvalue. Accordingly, a molecular image distributed about an attentionpixel can be identified with high accuracy by matching the molecularimage with a template in a divided pixel region even when a part of themolecular image overlaps another molecular image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic configuration diagram of a microscope device 1according to a preferred embodiment of the present invention.

FIG. 2 are conceptual views showing examples of templates prepared by atemplate preparation unit 15 of FIG. 1.

FIG. 3 is a graph showing an example of a distribution of luminancevalues in the column direction in an object area of an object forcalculating an evaluation value by an evaluation value calculation unit17 of FIG. 1.

FIG. 4 are conceptual views showing examples of partial templatesregenerated by the evaluation value calculation unit 17 of FIG. 1.

FIG. 5 is a flowchart showing the operation of a template preparationprocessing by an image processing device 10 of FIG. 1.

FIG. 6 is a flowchart showing the operation of a super-resolution imagepreparation processing by the image processing device 10 of FIG. 1.

FIG. 7 are views showing image pictures of object areas to be processedby the image processing device 10 of FIG. 1.

FIG. 8 is a diagram showing a hardware configuration of a computer forexecuting a program stored in a recording medium.

FIG. 9 is a perspective view of the computer for executing a programstored in a recording medium.

FIG. 10 is a flowchart showing the operation of a super-resolution imagepreparation processing by the image processing device 10 according to amodification of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments of an image processing device,imaging device, microscope device, image processing method, and imageprocessing program according to the present invention will be describedin detail with reference to the drawings. Also, the same orcorresponding parts will be denoted with the same reference signs in thedescription of the drawings, and overlapping description will beomitted.

FIG. 1 is a schematic configuration diagram of a microscope device 1according to a preferred embodiment of the present invention. Themicroscope device 1 shown in the same figure is a device for identifyingthe location of molecules such as fluorescent molecules or luminescentmolecules in a sample such as living cells, and is constructed with animage processing device 10 for processing a two-dimensional pixel imagepicture obtained by imaging the sample, a microscope 30 for obtaining anoptical image of the sample, a relay lens 50 for receiving an opticalimage obtained by the microscope 30, and an imaging element 40, such asa CMOS sensor or a CCD sensor, for taking an optical image of the samplefocused by the relay lens 50 and generating an optical image (imagepicture).

The microscope 30 has a sample stage 31 on which a sample is placed, anillumination light source 32 for irradiating the sample withillumination light, and an objective lens 33 which is an optical systemfor observation of the sample. The relay lens 50 is disposed between themicroscope 30 and the imaging element 40.

The image processing device 10 includes an image processing section 11which is a data processor, such as a server device and a personalcomputer, provided with a CPU and for performing processing and controlby software, an input unit 12 for inputting data to the image processingsection 11 from the outside, and a display unit 13 such as a display foroutputting data processed by the image processing section 11. The imageprocessing section 11 includes, as functional components, an opticalimage storage unit 14, a template preparation unit (template preparingmeans) 15, an area setting unit 16, an evaluation value calculation unit(evaluation value calculating means) 17, a molecular locationidentification unit (molecular location identifying means) 18, amolecular location storage unit 19, a difference image generation unit20, and an output image generation unit 21. Further, the templatepreparation unit 15 consists of a template setting unit 15 a, a templateinformation storage unit 15 b, and a divided region setting unit 15 c.

Hereinafter, the function of the components of the image processingsection 11 will be described in detail.

The optical image storage unit 14 is an information storage means fortemporarily storing an optical image generated by the imaging element40. The template setting unit 15 a reads out of the optical imagestorage unit 14 a single-molecule optical image obtained by the imagingelement 40 based on a model sample containing a single fluorescentmolecule, and prepares, from the single-molecule optical image, atemplate which is a two-dimensional image composed of pixels of M rowsand M columns (M is an integer not less than 3) corresponding to asingle-fluorescent-molecule model. Because single-molecule fluorescencegenerally has a two-dimensional Gaussian distribution, the templateprepared by the template setting unit 15 a results in image data of aluminance corresponding to a two-dimensional Gaussian distribution in apixel region corresponding to a circular region or substantiallycircular region within the range of M rows and M columns. Such acircular region or substantially circular region corresponds to theshape of the single-fluorescent-molecule model. FIG. 2(a) shows aconceptual view showing an example of a template T1 prepared with 5 rowsand 5 columns, and in the rectangular image of 5 rows and 5 columns inthe same figure, the shaded part corresponding to a circular region C1indicates being pixels for which significant values corresponding to atwo-dimensional Gaussian distribution have been set as luminance values.The “pixel with a significant value” referred to here means a pixel thatshows luminance, i.e. light emission, of a predetermined value (forexample, a zero value) or more in terms of a luminance value. Thetemplate information storage unit 15 b stores a template generated bythe template setting unit 15 a.

The divided region setting unit 15 c reads a template generated by thetemplate setting unit 15 a out of the template information storage unit15 b, prepares partial templates for which the template is divided intoa plurality of ratios defined in advance, and stores those partialtemplates in the template information storage unit 15 b. When describedin detail, the divided region setting unit 15 c, when a template isgenerated corresponding to the circular region C1, based on thetemplate, prepares a partial template as a pixel group corresponding toa fan shape having a predetermined central angle. Here, thepredetermined central angle is a central angle with an angle of α×360°obtained by multiplying 360°, which is a central angle of the circularregion C1 forming a template, by α (α is a number more than 0 and lessthan 1). The divided region setting unit 15 c prepares a plurality ofpartial templates different in the value of α. For example, as shown inFIGS. 2(b), (c), (d), and (e), the divided region setting unit 15 cprepares partial templates T2, T3, T4, and T5 having pixelscorresponding to fan-shaped regions C2, C3, C4, and C5 having centralangles obtained by multiplying 360°, which is a central angle of thecircular region C1, by ¼, ⅓, ½, and ⅔, respectively. The thus preparedpartial templates T2, T3, T4, and T5 are set to ones corresponding tothe fan-shaped regions C2, C3, C4, and C5 whose central angles are setto 90 degrees, 120 degrees, 180 degrees, and 240 degrees, respectively.

The area setting unit 16 reads an optical image of an object of amolecular location identification processing out of the optical imagestorage unit 14, and sets an object area of a template matchingprocessing for the optical image. That is, the area setting unit 16sets, corresponding to a pixel size of M rows and M columns being thesize of a single-fluorescent-molecule model, a rectangular object areaof 2×M rows and 2×M columns. Then, every time a template matchingprocessing is repeated for one optical image, the area setting unit 16changes object area in the optical image. For example, the area settingunit 16 sets the object area of 2×M rows and 2×M columns in the opticalimage by shifting by one pixel each time. In addition, the area settingunit 16 may set, as the object area of a template matching processing,an area corresponding to a circular region having a size of thesingle-fluorescent-molecule model, and may set the area with a size ofβ×M rows and β×M columns (β is a number more than 1).

The evaluation value calculation unit 17 selects an attention pixel inthe object area set by the area setting unit 16, and calculates, by useof a partial template corresponding to fan shapes rotated by apredetermined angle (angle larger than 0 degrees) each about theattention pixel, for every plurality of attention pixels included in theobject area, an evaluation value for evaluating luminance matchingbetween the optical image and the partial template. At this time, theevaluation value calculation unit 17 sets a using partial template basedon a luminance level of the attention pixel. That is, the evaluationvalue calculation unit 17 selects the central angle of the region C2,C3, C4, or C5 corresponding to a partial template according to theluminance value of the attention pixel, and reads a partial templatecorresponding to the selected region C2, C3, C4, or C5 out of thetemplate information storage unit 15 b.

FIG. 3 shows an example of a distribution of luminance values in thecolumn direction in an object area of an object for calculating anevaluation value by the evaluation value calculation unit 17. When theluminance values in the object area are thus distributed from theminimum value V_(LMIN) to the maximum value V_(LMAX), the evaluationvalue calculation unit 17 selects the partial template T5 (FIG. 2(e))whose central angle is 240 degrees for an attention pixel whoseluminance value is in a range R₁ from the minimum value V_(LMIN) to 30%between the minimum value V_(LMIN) and the maximum value V_(LMAX). Themaximum value V_(LMAX) is set to the maximum luminance determined afterexcluding 0.5% of highest luminance values from those of all pixels inthe object area so as not to include a noise component. Moreover, theevaluation value calculation unit 17 selects the partial template T4(FIG. 2(d)) whose central angle is 180 degrees for an attention pixelwhose luminance value is in a range R₂ of 30% to 50% between the minimumvalue V_(LMIN) and the maximum value V_(LMAX). Moreover, the evaluationvalue calculation unit 17 selects the partial template T3 (FIG. 2(c))whose central angle is 120 degrees for an attention pixel whoseluminance value is in a range R₃ of 50% to 70% between the minimum valueV_(LMIN) and the maximum value V_(LMAX). Further, the evaluation valuecalculation unit 17 selects the partial template T2 (FIG. 2(b)) whosecentral angle is 90 degrees for an attention pixel whose luminance valueis in a range R₄ from 70% between the minimum value V_(LMIN) and themaximum value V_(LMAX) to the maximum value V_(LMAX). The reason forthus setting the size of the central angle of the partial templateaccording to the luminance value is as follows. That is, it is highlylikely in a high-luminance location that molecules exist overlappingeach other and a part to match a template is limited to be narrow, whileit is unlikely in a low-luminance position that molecules existoverlapping each other and a part to match a template can be widelysecured.

Then, the evaluation value calculation unit 17, when calculating anevaluation value by use of a selected partial template for attentionpixels, overlays the partial template centered on the selected attentionpixel, calculates the sum of squared differences (SSDs) in luminancebetween the pixels in the object area around the attention pixels andthe partial template, and calculates as an evaluation value a valueobtained by normalizing the sum of squared differences by the luminanceof the attention pixel. Further, the evaluation value calculation unit17 regenerates a partial template corresponding to shapes of theselected partial template rotated by a predetermined angle each, andcalculates evaluation values in the same manner as the above by use ofthe regenerated partial template. When described in detail, theevaluation value calculation unit 17, when having selected the partialtemplate T2 (FIG. 4(a)) corresponding to the fan-shaped region C2 whosecentral angle is 90 degrees, regenerates partial templates by rotatingthe partial template by steps of an angle (for example, 360 degrees/16pixels=22.5 degrees in the case of a template of 5 rows and 5 columns)of the partial template divided by the number of pixels in thecircumferential direction. However, the angle steps by which the partialtemplate is rotated is not limited to an angle obtained by dividing bythe number of pixels in the circumferential direction, and may bearbitrary predetermined angle steps (for example, one degree each time)that are set in advance. FIGS. 4(b) to (f) show partial templates T21 toT25 regenerated corresponding to regions C21 to C25 obtained by rotatingthe fan-shaped region C2 about its center as a rotation center by 22.5degrees, 45 degrees, 292.5 degrees, 315 degrees, and 337.5 degrees,respectively. Further, the evaluation value calculation unit 17calculates evaluation values by use of the selected and regeneratedplurality of partial templates, and then determines a value with thehighest degree of matching, that is, the minimum calculated evaluationvalue, as an evaluation value of the attention pixels. In addition, theevaluation value calculation unit 17 may determine an average value ofevaluation values calculated for the plurality of partial templates asan evaluation value of the attention pixels.

The molecular location identification unit 18 targets evaluation valuescalculated for a plurality of object areas included in the optical imageto identify an evaluation value of a high degree of matching, andidentifies the position of attention pixels having the identifiedevaluation value as a molecular location in the optical image. Theidentification of an evaluation value of a high degree of matching maybe performed by comparison with a preset threshold, or may be byselecting an evaluation value of the highest degree of matching in theobject area. Moreover, in the case of simultaneously identifying aplurality of molecular locations in the optical image, the molecularlocation identification unit 18 may select a plurality of molecularlocations so that the distance between the molecular locations exceeds apredefined distance (for example, the size of one molecule). Themolecular location identification unit stores the identified molecularlocation in the molecular location storage unit 19 every time ofidentification.

The difference image generation unit 20 excludes, from the optical imageprocessed by the area setting unit 16, the evaluation value calculationunit 17, and the molecular location identification unit 18, withreference to the molecular location identified by the molecular locationidentification unit 18, the luminance value corresponding to a templatecorresponding to a single-fluorescent-molecule model. Specifically, thedifference image generation unit 20, in the optical image, around apixel corresponding to the molecular location, from the luminance valueof each of the plurality of pixels corresponding to a template,subtracts the luminance value at a corresponding position in thetemplate arranged centered on the molecular location. The differenceimage generation unit 20 outputs the optical image subjected tosubtraction processing to the area setting unit 16, and causes thetemplate matching processing and molecular location identificationprocessing for the optical image to be repeated.

The output image generation unit 21 generates a super-resolution imagewhich is an output image indicating a molecular location stored in themolecular location storage unit 19, and outputs the image to the displayunit 13. The super-resolution image may be output superimposed on anoptical image stored in the optical image storage unit 14, or may beoutput as a sole image indicating a molecular location.

Hereinafter, the operation of the image processing device 10 will bedescribed, while an image processing method according to the presentembodiment will be explained in detail. FIG. 5 is a flowchart showingthe operation of a template preparation processing by the imageprocessing device 10. FIG. 6 is a flowchart showing the operation of asuper-resolution image preparation processing by the image processingdevice 10. FIG. 7 are views showing image pictures of object areas to beprocessed by the image processing device 10.

Referring to FIG. 5, when preparing a super-resolution image from anoptical image of a sample of a processing object, a template preparationprocessing is executed in advance by the image processing device 10prior thereto. First, a single-molecule optical image is obtained by theimaging element 40 based on a model sample containing a singlefluorescent molecule, and the obtained single-molecule optical image isstored in the optical image storage unit 14 (step S01). Next, by thetemplate setting unit 15 a, the single-molecule optical image is readout of the optical image storage unit 14, and a template which is atwo-dimensional image of M rows and M columns (M is an integer not lessthan 3) is prepared from the single-molecule optical image (step S02).The prepared template is stored in the template information storage unit15 b. Further, by the divided region setting unit 15 c, a partialtemplate which is a pixel group corresponding to a fan shape having apredetermined central angle (a (a is a number more than 0 and less than1)×360°) is prepared from the template (step S03). The partial templateis prepared as a plurality of partial templates different in centralangles, and the prepared plurality of partial templates are stored inthe template information storage unit 15 b.

Referring to FIG. 6, after completion of the above-described templatepreparation processing, a super-resolution image preparation processingfor preparing a super-resolution image from an optical image of a sampleof a processing object is started by the image processing device 10.When described in detail, first, an optical image is obtained by theimaging element 40 based on an object sample, and the obtained opticalimage is stored in the optical image storage unit 14 (step S101). Then,by the area setting unit 16, an object area (FIG. 7(a)) of β×M rows andβ×M columns (β is a number more than 1) is set in plural numbers in theoptical image, and an object area-based molecular luminance map isprepared (step S102). The molecular luminance map is prepared as datafor which a total of luminance values of a single-molecule region(single-molecule-sized square region) around each pixel in the objectarea is mapped onto each pixel. Then, by the area setting unit 16, basedon the molecular luminance map, a pixel having a total of luminancevalues not less than a preset threshold is selected as an attentionpixel (step S103, FIG. 7(b)). For example, by the area setting unit 16,the pixels shown by shading in FIG. 7(b) are selected by being narroweddown as attention pixels.

Further, by the evaluation value calculation unit 17, one of thenarrowed down attention pixels is selected (step S104). Next, by theevaluation value calculation unit 17, the central angle of a partialtemplate to be used for an evaluation value calculation is set accordingto the luminance value of the attention pixel, and a partial templatecorresponding to the set central angle is read out of the templateinformation storage unit 15 b (step S105). Then, by the evaluation valuecalculation unit 17, an evaluation value of the attention pixels iscalculated by using the read-out partial template (step S106). Further,by the evaluation value calculation unit 17, whether an attention pixelof an object for calculating an evaluation value remains is determined(step S107). As a result of determination, when there is a remainingattention pixel (step S107; YES), the process is returned to step S104to repeat the setting of a partial template and the calculation of anevaluation value. On the other hand, when there is not a remainingattention pixel (step S107; NO), by the molecular locationidentification unit 18, the position of a candidate pixel being acandidate for a molecular location is identified (step S108, FIG. 7(c)).For example, by the molecular location identification unit 18, theposition of the pixel shown by shading in FIG. 7(c) is identified as acandidate pixel.

Next, by the difference image generation unit 20, with reference to themolecular location identified by the molecular location identificationunit 18, the template corresponding to a molecular image of asingle-fluorescent-molecule model is subtracted from the optical image(step S109, FIG. 7(d)). Further, by the difference image generation unit20, whether there is a pixel (effective pixel) having a significantvalue in the optical image after subtraction is determined (step S110).As a result, when it is determined that there is an effective pixel(step S110; YES), the process is returned to step S102 to repeat theattention pixel narrowing-down processing and the molecular locationidentification processing for the optical image after subtraction (FIGS.7(e) and (f)). On the other hand, when it is determined that there isnot an effective pixel (step S110; NO), the molecular locationidentification processing is completed, and by the output imagegeneration unit 21, a super-resolution image is generated and outputbased on the candidate pixel(s) having been identified so far (stepS111).

Hereinafter, an image processing program for causing a computer tooperate as the image processing device 10 will be described.

The image processing program according to the embodiment of the presentinvention is provided, stored in a recording medium. As the recordingmedium, a recording medium such as a floppy (registered trademark) disk,a CD-ROM, a DVD, or a ROM, a semiconductor memory, or the like isexemplified.

FIG. 8 is a diagram showing a hardware configuration of a computer forexecuting a program stored in a recording medium, and FIG. 9 is aperspective view of the computer for executing a program stored in arecording medium. Examples of the computer include various dataprocessors, such as server devices and personal computers, provided withCPUs and for performing processing and control by software.

As shown in FIG. 8, the computer 130 includes a reading device 112 suchas a floppy (registered trademark) disk drive device, a CD-ROM drivedevice, or a DVD drive device, a working memory (RAM) 114 in which anoperating system is resident, a memory 116 for storing a program storedin a recording medium 110, a display device 118 such as a display, amouse 120 and a keyboard 122 being an input device, a communicationdevice 124 for performing transmission and reception of data etc., and aCPU 126 for controlling execution of a program. The computer 130, whenthe recording medium 110 is inserted in the reading device 112, canaccess an image processing program stored in the recording medium 110through the reading device 112, and is enabled by said image processingprogram to operate as the image processing device 10 of the presentembodiment.

As shown in FIG. 9, the image processing program may be provided over anetwork as a computer data signal 141 superimposed on a carrier. In thiscase, the computer 130 can store an image processing program received bythe communication device 124 in the memory 116, and execute said imageprocessing program.

According to the microscope device 1 described above, a partial templatedivided from a template included in pixels of M rows and M columns isprepared, and by use of a partial template corresponding to a shaperotated about an attention pixel selected from an optical image of anobject sample, matching between the partial template and optical imageis evaluated, and a molecular location is identified based on theobtained evaluation value. Accordingly, a molecular image distributedabout an attention pixel can be identified with high accuracy bymatching the molecular image with a template in a divided pixel regioneven when a part of the molecular image overlaps another molecularimage. Specifically, an image having a shape corresponding to a circularimage is generated as a template, an image corresponding to a fan shapeis generated from the circular image as a partial template, and thedegree of matching is evaluated by means of the partial template thateasily fits into an edge portion of a molecular image, and thus thecentral position of a molecule can be identified with a higher accuracyeven for an optical image where a plurality of molecular images areoverlapping. That is, by using the partial template corresponding to afan shape, an edge portion of a single molecule can be found out withhigh accuracy to identify based thereon the central position of themolecule.

Moreover, it is preferable for the partial template to have pixelscorresponding to a fan shape having a central angle for which a centralangle of the template is multiplied by a (a is a number less than 1 andmore than 0). Accordingly, when the template is circular orsubstantially circular, a partial template having an appropriate shapecan be prepared.

Moreover, it is preferable for the partial template to be prepared as aplurality of partial templates different in central angles. Accordingly,different partial templates can be used according to the situation.

Moreover, because the central angle of a partial template is selectedcorresponding to the pixel value of an attention pixel of an object forevaluating matching by the evaluation value calculation unit 17, as aresult of the division ratio of a partial template being adjustedaccording to the degree of overlapping of a plurality of molecules in anoptical image, matching between a molecular image and a partial templatecan be appropriately evaluated. Consequently, a molecular location canbe identified with high accuracy.

It is preferable, when the evaluation value calculation unit 17calculates an evaluation value by use of a partial templatecorresponding to shapes rotated by a predetermined angle each about aselected attention pixel, that the predetermined angle is an angleobtained by dividing by the number of pixels in the circumferentialdirection of a pixel group of M rows and M columns corresponding to atemplate. At this time, the number of pixels in the circumferentialdirection of a pixel group of M rows and M columns is 4×M−4 pixels.Accordingly, overlapping of the partial templates can be reduced, and acalculation of an evaluation value can be efficiently performed.

Moreover, because an attention pixel is selected out of the pixelsincluded in an optical image based on the pixel values by the evaluationvalue calculation unit 17, a processing for calculating an evaluationvalue and identifying a molecular location is performed for only theattention pixels with a high probability that a molecule exists. As aresult, the processing load of the device can be reduced and a molecularlocation is identified with high accuracy.

It is more preferable that an object area to be evaluated by theevaluation value calculation unit 17 is β×M rows and β×M columns (β is anumber more than 1) when a template of M rows and M columns is used.Even when a plurality of attention pixels having high evaluation valuesexist in a narrow range, a molecular location can be identified withhigh accuracy.

In addition, the present invention is not limited to the embodimentmentioned above. For example, the image processing device 10 may form animaging device such as a camera unit integrated with the imaging element40.

Moreover, when a template and a partial template are prepared by thetemplate setting unit 15 a and the divided region setting unit 15 c,these may be prepared based on various parameters such as a centralangle input from the outside by the input unit 12, and a luminance rangefor determining a central angle of the partial template.

Moreover, after identifying a molecular location for an optical image bythe image processing section 11, a molecular location may bere-identified for pixels for which a candidate pixel corresponding to amolecular location is divided into a plurality of divided regions. FIG.10 is a flowchart showing the procedure for a super-resolution imagepreparation processing by the image processing section 11 in this case.

As shown in the same figure, when the super-resolution image preparationprocessing is started, in the same manner as in step S101 to step S108of FIG. 6, the position of a candidate pixel in an optical image isidentified (step S201, step S202). Next, each pixel of the optical imageis divided into sub-pixels (divided regions) of 10 rows and 10 columnsby the area setting unit 16 (step S203). Then, for the sub-pixelsincluded in the candidate pixel in the divided optical image, in thesame manner as in step S102 to step S108, evaluation values arecalculated, a detailed position of the candidate pixel is identified insub-pixels based on the evaluation values (step S204). Here, as atemplate and partial template to be used, ones re-prepared correspondingto the resolution of the divided optical image and the size of asingle-fluorescent-molecule model are used. Then, by the differenceimage generation unit 20, the template corresponding to thesingle-fluorescent-molecule model is subtracted from the optical imagewith reference to the position of the candidate pixel identified (stepS205). Further, by the difference image generation unit 20, whetherthere is an effective pixel in the optical image after subtraction isdetermined (step S206). As a result, when it is determined that there isan effective pixel (step S206; YES), the process is returned to stepS202 to repeat the molecular location identification processing for thenon-divided and divided optical images after subtraction. On the otherhand, when it is determined that there is not an effective pixel (stepS206; NO), the molecular location identification processing iscompleted, and by the output image generation unit 21, asuper-resolution image is generated and output based on the candidatepixel(s) having been identified so far (step S207). According to such asuper-resolution image preparation processing, a molecular location inthe optical image can be identified in greater detail.

Moreover, as the optical image of a processing object of the imageprocessing device 10, an original image obtained by the imaging element40 may be used, and an optical image for which the resolution of anoriginal image is degraded by downsampling may be used when theresolution of the original image is relatively high.

Moreover, the image processing device 10 may set an object area aftercalculating an evaluation value by performing template matching toattention pixels in the optical image, and identify a pixel with thehighest degree of matching in the object area as a molecule location.

That is, the evaluation value calculation unit 17 selects an attentionpixel out of a plurality of pixels composing an optical image, andcalculates, by use of a partial template corresponding to fan shapesrotated by a predetermined angle (angle larger than 0 degrees) eachabout the attention pixel, an evaluation value for evaluating luminancematching between the optical image and the partial template. Theevaluation value calculation unit 17 calculates a comparative evaluationvalue of the selected attention pixel based on a plurality of calculatedevaluation values.

Then, the molecular location identification unit 18 stores as amolecular location the position of an attention pixel where the degreeof matching indicated by a comparative evaluation value is the highestout of a plurality of attention pixels in the object area set by thearea setting unit 16. Accordingly, even when attention pixels with highdegrees of matching are close, a molecular location can be identifiedwith high accuracy.

Here, it is also preferable that the template has pixels correspondingto a circular image, and the partial template has pixels correspondingto a fan shape having a central angle for which a central angle of thecircular image is multiplied by a (a is a number less than 1 and morethan 0). In this case, as a result of the degree of matching beingevaluated by means of the partial template that easily fits into an edgeportion of a molecular image, the central position of a molecule can beidentified with a higher accuracy.

Moreover, it is preferable that the evaluation value calculating meansselects the predetermined ratio corresponding to the partial template,corresponding to a pixel value of the attention pixel of an object forevaluating the matching. By including such an evaluation valuecalculating means, the division ratio of a partial template is adjustedaccording to the degree of overlapping of a plurality of molecules, sothat matching between a molecular image and a partial template can beappropriately evaluated. Consequently, a molecular location can beidentified with high accuracy.

Further, it is also preferable that the evaluation value calculatingmeans selects, out of pixels included in the image picture, theattention pixel of an object for calculating the evaluation value basedon pixel values of the pixels. By having such a constitution, aprocessing for calculating an evaluation value and identifying amolecular location is performed for only the attention pixels with ahigh probability that a molecule exists, so that the processing load ofthe device can be reduced and a molecular location is identified withhigh accuracy.

Still further, it is also preferable that the evaluation valuecalculating means divides the attention pixel identified as themolecular location by the molecular location identifying means into aplurality of divided regions, and recalculates the evaluation value forevery plurality of divided regions by use of the partial template, andthe molecular location identifying means re-identifies the molecularlocation in the image picture based on the evaluation value calculatedfor the plurality of divided regions in the attention pixel. Accordingto such a constitution, a molecular location in the image picture can beidentified in greater detail.

1-9. (canceled) 10: An image processing device for identifying amolecular location based on an image picture of a sample, comprising: amemory configured to store at least one image picture captured by acamera, the image picture including a plurality of pixels; a processorcommunicatively coupled to the memory, the processor configured to:prepare a template corresponding to a molecular model, the templatebeing a two-dimensional image; prepare a plurality of partial templatesfrom the template, wherein each partial template corresponds to acircular sector shape having a central angle; obtain the image picture;select at least one pixel out of the pixels included in the imagepicture for which an evaluation value is to be calculated; set a centralangle of a partial template based on a luminance value of the selectedpixel; select a partial template out of the prepared plurality ofpartial templates according to the set central angle; calculate anevaluation value of the selected pixel using the selected partialtemplate; and identify the molecular location in the image picture basedon the calculated evaluation value. 11: The image processing deviceaccording to claim 10, wherein the template comprises pixelscorresponding to a circular image, and the central angle is set bymultiplying a central angle of the circular image by α (α is a numberless than 1 and more than 0). 12: The image processing device accordingto claim 11, wherein the processor is configured to select apredetermined ratio corresponding to the number a based on the luminancevalue of the selected pixel. 13: The image processing device accordingto claim 10, wherein the processor is configured to identify theselected pixel as the molecular location, to divide the selected pixelidentified as the molecular location into a plurality of dividedregions, to recalculate the evaluation value for each of the dividedregions by use of the partial template, and to re-identify the molecularlocation in the image picture based on the evaluation value calculatedfor the plurality of divided regions in the selected pixel. 14: Animaging device comprising: the image processing device according toclaim 10; and the camera configured to capture the image picture. 15: Amicroscope device comprising: the image processing device according toclaim 10; the camera configured to capture the image picture; and anoptical system configured to generate an optical image of the sample andto provide the optical image for the camera. 16: An image processingmethod for identifying a molecular location based on an image picture ofa sample, comprising: storing at least one image picture captured by acamera into memory, the image picture including a plurality of pixels;preparing a template corresponding to a molecular model, the templatebeing a two-dimensional image; preparing a plurality of partialtemplates from the template, wherein each partial template correspondsto a circular sector shape having a central angle; obtaining the imagepicture; selecting at least one pixel out of the pixels included in theimage picture for which an evaluation value is to be calculated; settinga central angle of a partial template based on a luminance value of theselected pixel; selecting a partial template out of the preparedplurality of partial templates according to the set central angle;calculating an evaluation value of the selected pixel using the selectedpartial template; and identifying the molecular location in the imagepicture based on the calculated evaluation value. 17: A non-transitorycomputer-readable medium containing program instructions for causing acomputer to perform a method, in an image processing device foridentifying a molecular location based on an image picture of a sample,of: storing at least one image picture captured by a camera into memory,the image picture including a plurality of pixels; preparing a templatecorresponding to a molecular model, the template being a two-dimensionalimage; preparing a plurality of partial templates from the template,wherein each partial template corresponds to a circular sector shapehaving a central angle; obtaining the image picture; selecting at leastone pixel out of the pixels included in the image picture for which anevaluation value is to be calculated; setting a central angle of apartial template based on a luminance value of the selected pixel;selecting a partial template out of the prepared plurality of partialtemplates according to the set central angle; calculating an evaluationvalue of the selected pixel using the selected partial template; andidentifying the molecular location in the image picture based on thecalculated evaluation value. 18: The image processing device accordingto claim 10, wherein the processor is configured to set a small centralangle of the selected partial template when the luminance value of theselected pixel is high.